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ser ' es °f guides is designed to help school dis- 
trict leaders address one of the toughest challenges 
mfi in American education: dropout rates of 30 percent 
nationwide, 50 percent in many big cities, and 60 per- 
cent or more in the lowest-performing schools. 

The good news is that several large urban districts, 
intent on raising graduation rates and increasing 
college readiness, have been strategically address- 
ing these challenges for the past several years. By 
better understanding the needs of their students, 
district leaders have created a mix of school designs 
and programs— a portfolio of educational options. 
This series shares their strategies, offers advice, and 
provides practical tools to help leaders break down 
this seemingly intractable crisis into a series of more 
manageable steps. 

The approaches documented in these guides are 
promising and have some evidence of success. But 
the efforts remain a work in progress whose long- 
term impact will not be known for several more 
years. 

The first guide in the series [Leadership Guide 1, an 
overview for decisionmakers, describes in abbrevi- 
ated form how districts can: 

■ pinpoint how students are progressing and which 
students, by name, are most likely to struggle in 
school and drop out 

■ introduce some high-leverage strategies to get 
students back on track for a diploma 

■ identify the mix of school choices and programs 
that will prepare more students for colleges and 
careers 



The second guide [ implementation Guide I offers a 
more detailed examination of the six key questions 
that districts are addressing: 

■ How are your students progressing— and which 
are struggling? 

■ What kind of school choices do you provide to 
meet diverse student needs— and how well are 
those schools and programs performing? 

■ How will you manage a change process, inviting 
multiple stakeholders inside and outside the 
system to make the kinds of changes that the data 
suggest are needed? 

■ How can you strengthen your portfolio of options? 

■ How will you provide support to schools? 

■ What policy changes are needed? 

The third guide [Analyst Guidel includes tools for 
data analysts to drill down into the data and use 
their findings to arm school leaders with actionable 
information (online only). 

These guides build on the first phase of education 
work of the Bill & Melinda Gates Foundation— helping 
districts build a portfolio of smaller, theme-based 
schools. They respond to multiple requests from poli- 
cymakers and educators who asked us to share what 
we have learned in a form that they can use in their 
own communities. Information is drawn from Atlanta, 
Boston, Charlotte-Mecklenburg, Chicago, Dallas, 
Portland, and New York City and their partnerships 
with the Bridgespan Group, Boston Consulting Group, 
McKinsey & Company, Education Resource Strategies, 
and The Parthenon Group. 
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Based on lessons learned from this phase, the foun- 
dation is now focused on three areas in which we are 
uniquely positioned to make a large-scale impact: 

supporting the development and implementation 
of college-ready standards, as well as tools for 
students and teachers to implement them 

empowering excellent teachers 

finding innovative ways to support the next 
generation of school models 

In light of the proposed criteria for education 
stimulus funding through the U.S. Department of 
Education’s Race to the Top, the advice offered in 
these guides is particularly relevant and timely for 
any district committed to establishing data systems 
to track student achievement, turning around low- 
performing schools, and developing the right mix 
of offerings for each and every one of its students— 
and the thousands of others who share the dream 
of a better life. 



1. Use Data To Inform Key Decisions g 

2. Conduct Three Key Analyses § 

3. Find Out Which Students Are 

Struggling 0 

4. Strengthen Effective Options [|§ 

5. Calculate the Financial Impact m 

6. Use Data To Drive Change 



The guides are intended to be just that — guides, not instruction manuals. 

You will have your own answers to the questions found here and can browse 
quickly through whole sections to learn how your experience matches that of 
other districts. Likewise, you can approach this work in a different sequence, 
beginning with building community support for change or assessing the 
effectiveness of the mix of schools and programs you have now. 





B ry district uses data. The most important 

ion is how and to what end those data are used. 

uroan districts are using data— or should be— to 
address the challenge of students dropping out of 
school. Numbers are crunched in databases with 
hundreds of thousands of student records to know 
the size of the dropout problem and the track record 
of schools in stemming the loss. Data tools are 
employed to calculate the cost of various options, 
from closing schools and opening new ones to 
providing interventions such as evening school. 
Performance data are analyzed to evaluate the suc- 
cess of the district’s portfolio of schools. In sum, 
data provide the foundation for the myriad district 
decisions involved in matching student needs to 
school and program choices. 

It is quite likely that leaders in your district are 
intent on using data to inform some hard choices 
ahead — but they need your help. The analyses and 
charts in this guide illustrate the work you need to 
do to answer the questions they encountered in the 
Leadership and Implementation guides. If you are not 
familiar with these guides, then the first step is to 
spend some time with them. 

After a review of the other guides and an examina- 
tion of this one, you will be in a better position to 
move to the next step— assessing whether your 
district has the requisite system for data collec- 
tion and analysis, and if not, where you will pull the 
information from. You can then move forward with 
data analysis or invest resources to increase the 
internal capacity for analysis or request outside 
help as needed. (For an example of an RFP for data 
support, see Jobs for the Future, Bringing Off-Track 
Youth into the Center of High School Reform, pages 
50-55.) Several of the districts cited in these guides 
turned to consultants for support for the initial 
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round of data analysis to identify which indicators 
were most predictive of future dropout rates and 
which schools were beating the odds. Since this 
guide depicts the analyses conducted, you may need 
less support from consultants as a result. 

However, an assignment of this magnitude is not 
one you will want to undertake alone. After consult- 
ing with district leadership, you can form a team of 
school and central office administrators to generate 
questions, confirm the validity of the analysis, and 
ensure that findings are clearly presented. A budget 
analyst on the team will be valuable, for example, 
when it is time to calculate the cost of various 
school and program options and their return on 
investment. 

Every 26 seconds another student gives up on 
school. Three students in 10 fail to graduate from 
public high school with their class, according to 
the Alliance for Excellent Education. Good data 
presented well create a picture of a problem that 
everyone can see. Use the tools and charts in this 
guide as models to generate and present the data 
that your district needs to better serve students. 




You can find more tools, charts, and 
graphs in the Suj^Ument to tjie Analyst 
Guide . Look for this icon for a description 
of these additional resources. 
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Examine student progress, school/program strengths, and district investments. 

The table below identifies key questions to be answered and the decisions that can be made once good data are 
available. 





Major Analyses 


Questions Answered 


Decisions/Actions Enabled 


Student 


Track Student 


Which students are struggling and dropping out? 


Dispelling myths about the nature of dropouts 




Progress 


Which student performance indicators predict 
falling off track or dropping out? 


Defining at-risk and off-track indicators to 
monitor student progress and trigger early 
intervention 


School 

and 


Create a Mix 
of Schools and 


Are there systematically over- and under- 
performing school models? 


Phasing out ineffective schools and models 


District 


Programs 


What are the potential improvements in 
graduation rates from different options? 


Replacing ineffective schools with models 
that maximize the probability of success 






Do we currently offer effective options for off- 
track students? 


Prioritizing options— school models and pro- 
grams— to match demonstrated student need 






What options “best” serve different groups of 
struggling students? 


Designing admissions and programs for 
recuperative models 






How does the supply of seats and options 
match the level of demand for services? 


Planning for new construction/new seats 




Optimize 

Resources 


What resources are currently allocated to 
which students and schools? 


Aligning resources based on student and 
school need 






What additional investments are necessary to 
meet district needs? 


Prioritizing resources to most effective 
programs 






What are the expected returns/results from 
investments? 


Organizing people and time in ways that 
facilitate high performance 
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Be clear on the goals of the analysis. 

Understand what your district’s leaders want to 
learn— the focus of their strategy and the data 
required for specific decisions. Are you creating 
an initial diagnostic to see how well your schools 
are serving students now, or are you beginning 
the analysis with a focus on finding the potential 
dropouts in your schools? The guide includes data 
tools for each of the three kinds of analysis, but 
your district’s plan of attack determines where 
you start. You can help determine the extent of the 
analysis, advising decisionmakers on the tradeoff 
between cost and depth of analysis. For example, 
your district may decide to identify and support just 
those students who are severely off track toward 
graduation rather than including those who are 
slightly behind in credits. 



a 

Know the district's target for graduation. 

While every district wants 100 percent of its students 
to graduate, setting a target of 100 percent will not 
give you the focus you need to make changes. Some 
schools in your district may be graduating 80 per- 
cent or more of their students, while others graduate 
considerably fewer. You will want to work with the 
schools that need the most attention. Consider an 
initial target of 60 percent and determine how many 
schools are failing to hit this target. Nowyou can 
focus your efforts on the students in these schools. 
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Know which indicators will be used to measure progress. 

Outcome indicators include measures such as graduation rate and the number of students attending college. 
Leading indicators measure behaviors or characteristics that tend to predict future events or behavior. For example, 
being behind in credits earned and failing at least one core course by the end of 9th grade could be a leading indica- 
tor of dropping out. These indicators will dictate the data you need to collect. 



Sample Outcome Indicators 



Sample Leading Indicators 



Early at-risk indicators: demographic 

■ Special education status (especially students in 
substantially separated classrooms) 

■ Age upon entering high school 

■ English language learner status 
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■ Graduation rates 

■ Standardized test scores 

■ College admissions rates 

■ College success rates 



Early at-risk indicators: performance 

■ Low-to-failing grades in core courses (math and 
English) in 8th grade 

■ Poor standardized test scores in 8th grade 

■ Consistent absenteeism 



■ Behind in credits accumulated during first year in 
high school 

■ Course failures (especially core courses such as 
math and English) 

■ Consistent absenteeism 



9th grade at-risk indicators 



Collect and prepare the data. 

This step requires defining and collecting the necessary data, cleaning the data, and creating longitudinal cohorts 
of students. 



Collecting and Preparing Data 



Define and Collect 
Necessary Data 
Elements 


Detenmine what data ane nequined to penfonm an analysis and whene data elements exist acnoss 
distnict systems 


Clanify the definitions and calculations of key data elements (i.e. , gnaduation nates] 


Collecting necessany data elements may nequine sevenal weeks of wonk, depending on the fonmats, 
complexity, and dispanateness of distnict data systems 


Identify Data 
Inconsistencies 


Some metnics ane not consistently tnacked non input into systems acnoss all school types 


Definitions and metnics may be inconsistent acnoss schools. Common data inconsistencies include: 

■ Student cnedits eanned towand gnaduation 

■ Reason fon dischange/tnansfen between pnognams 

■ Attendance nates 


Create Longitudinal 
Cohorts of Student 
Data 


Detenmine nules fon which students ane in which cohont 


Fon each student, captune nelevant demognaphic and penfonmance metnics (e.g ., student 
attendance pnion to and aften entny to a specific necupenative school) 



Insight ▼ 

Most data are available in the district systems— the challenge lies with the complexity and accuracy of data collection, 
synthesis, and analysis. 




Defining What Data Are Needed and Making a Request 



Type of Data 


Example Fields 


Student 

Identification 

and 

Demographics 


■ Student ID ■ Fnee/Reduced-pnice lunch 

■ Genden ■ Home ZIP Code 

■ Binth date ■ Ethnicity 

■ ELL status ■ SPED status 


Enrollment 

Status 


Date of entny to schools ■ School ennolled 

Completion/withdnawal Date of exit fnom schools 


Academic 

Performance 


■ Counse numbens ■ Counse gnades 

■ Counse absences ■ Counse tandies 

■ Cnedits eanned ■ Standandized test nesults 


Student Behavior 


Attendance Discipline necond 


District 

Information 


■ Complete list of schools ■ School type fon each school 



Success Factor 

Gathering data can take significant amounts of time, so spending time at the 
beginning of the project and requesting all the data you may conceivably need can 
save time later. 



Key Steps 

1. Determine the types of analyses 
to be penfonmed and the data 
nequined. 

2. Put togethen a data nequest that 
outlines the data needed in a 
simple fonm. 

3. Request data fon multiple points 
duning the schoolyean, beginning 
with 8th gnade, if a longitudinal 
analysis is to be penfonmed to 
allow monitoning of students. 

4. Canefully check data aften 
annival to ensune inclusion of all 
fields and all dates nequested. 

Key Questions 

1. How many yeans of data ane 
needed to sufficiently penfonm 
the analysis? 

2. How long will it take to neceive 
data and what can be done 
duning the waiting time? 
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Structure the data sets. 

Decide whether to use linked data sets or a single comprehensive data set. Each has advantages and drawbacks. 
Linked data sets provide more flexibility but are more difficult to program. Single sets are simpler to create but 
require a powerful system to manage. 



OPTION 1— LINKED DATASETS 



Demographic Data Performance Data Enrollment Data 



Unique ID 



Unique ID 




Option 1 

Pros 

■ Requires minimal 
upfront wonk to fonmat 

■ Pnovides gneaten 
flexibility 

■ Relatively fast with an 
advanced database 
pnognam such as SAS 
on SPSS 



Cons 

■ Mone difficult to 
pnognam and use on a 
daily basis 

■ Mone difficult to keep 
tnack of whene all the 
nelevant data fields ane 
stoned 



OPTION 2— SINGLE, COMPREHENSIVE DATASET 



Demographic Data Performance Data Enrollment Data 




Option 2 

Pros 

■ Simplen way to have 
all necessany data in a 
single place 

■ Easiento conceptualize 
wonk 

■ Can be fasten fon 
pnognams such as 
Access that do not link 
data sets efficiently 



Cons 

■ Requines significant 
time upfnont to collapse 
all data sets into a 
single set 

■ Pnoduces a veny lange 
data set that can 
nequine a powenful 
computen to manage 





This slide is accessible online in the Supplement to 
the Analyst Guide: 

Creating a Project Timeline is a sample of a three- 
month timeline for launching the project, conduct- 
ing analysis, and engaging stakeholders. 
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reate a database with students 
in the most recent graduating 
class. 

Include fields for all demographics, 
such as date of birth, ethnicity, poverty 
status, English language learner (ELL) 
status, special education status and 
disability type; academic performance 
fields, such as credits earned and test 
scores; behavior data elements, such 
as attendance; and school(s) attended. 
Start your data collection with student 
history from 8th grade and continue 
through graduation. 

Key questions: 



Identify struggling students by all the 
indicators that might predict future 
dropouts. 

At-risk indicators can vary across districts but 
commonly include age at entering 9th grade, 
absenteeism, low-to-failing grades in core 
courses, special education or English proficiency 
status, and poor standardized test scores. 

Off-track indicators include course failures, 
behind in credits earned, and poor performance 
on achievement tests. 

Four indicators have been predictive of future 
dropout rates in multiple districts. While they 
may not be as meaningful in your district, it 
would be worthwhile to use these four: 



Should the cohort include students 
who transferred into the district after 
9th grade? 

Should the cohort include students 
who transfer out before graduating? 



Are there students who should 
be excluded from the cohort, e.g., 
severely developmentally delayed 
students? 



age 15orolderwhen entering 9th grade 

absent more than 10 days of fall semester of 
9th grade 

failed two or more courses fall semester of 
9th grade 

had fewer than five credits or failed two or more 
courses in the spring semester of 9th grade 



Some students will be in more than one group; 
e.g., an ELL student could be in a group of 
males, minority males, students who failed 
two or more courses, and students with high 
absenteeism. 
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Measure the accumulation of credits. 

A key predictor of future dropouts is the accumulation of credits for graduation. Once you have defined how many 
credits a student must earn to stay on track, use credits earned to gauge when students are off track. 

In the chart below, students who were .5-2 credits behind were defined as moderately off track, while those 
who were 2.5 to 6 credits behind were severely off track. 

Your ability to pull the data and produce charts that clearly display student progress will be invaluable to your 
district in planning interventions. While nearly 20 percent of 9th graders were already severely off track at the 
end of Year 1 , and even more by the end of Year 2, early interventions might have prevented such outcomes for 
these students. 

Key Steps 

1. Select a cross-sectional population of all 
students enrolled in high school at a given time- 
likely either the beginning or end of a school year. 

2. Apply the off-track definition to a// students 
enrolled [exclude students who have dropped 
out of the system). 

3. Group students based on severity of off-track sta- 
tus using some combination of years in high school 
and credits earned relative to being on pace. 

Key Questions 

1. At what point in time does the district measure 
the off-track population? 

2. How do the off-track students split based on 
levels of need? 

3. How has the size of the off-track population 
changed over time? 




Credit Status by Student Year, Spring 2007 



100% 



27.4K 24.9K 20. 2K 







17.4K Severely Off Track, 
2.5 to 6+ Credits 
Behind 

Moderately Off 
Track, 0.5 to 2 
Credits Behind 

On Track, with 

Some Failures 



On Track, 
No Failures 



Year 1 Year 2 Year 3 Year 4 
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Learn which indicators are most predictive of whether a student will drop out. 

Predictive indicators. Calculate the dropout rate for previous cohorts of students who fit the definition of your 
selected indicators, for example, students with many absences, ELL students, students off track for cred- 
its needed by 10th grade, African-American students, male students, or students with a D or F in math and 
English courses. Once this analysis is complete, your data will look similar to the chart below on the left. 

Comprehensive indicators. Now determine the size of any one group of dropouts in a historical cohort relative 
to the entire population of dropouts in that cohort, represented as a percentage. In the chart below on the right, 
students with many absences represent 47 percent of the dropouts in this cohort. 



100 % 



Predictive 

(% of Each Population Group 
Who Ultimately Drop Out) 



70% of all off-track 
students ultimately 
drop out 




100 % 



Comprehensive 
(% of All Dropouts in 
Each Population Group) 



82% of all dropouts 
were off-track 
students coming out 
of freshman year 




Key Definitions 

Predictive means the percentage of a population group that drops out of high school. 
Comprehensive means the percentage of all dropouts that belong to a population group. 



Key Steps 

1. Define the outcome metric and cohort to be used. 

2. Calculate the predictiveness of an indicator. 

3. Calculate the comprehensiveness of an indicator. 

4. Repeat for multiple demographics and early performance 
results to find a set of indicators that are collectively 
comprehensive. 



Calculation Example 

1. A district has 10,000 high school dropouts in a class. 
Three thousand students enter high school overage, 
of whom 1 ,500 drop out of school. 

Predictiveness: 

1,500 dropouts in an overage population of 3,000 = 50 percent 
Comprehensiveness: 

1,500 overage dropouts in a total dropout population of 
10,000 = 15 percent 
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Use what you have learned to decide where to focus attention with your current students. 

In the example on the previous page, off-track students made up 82 percent of the total dropout population. 

Based on this information— and what has been learned about off-track status as a predictor of dropouts in urban 
schools— a district would want to monitor and intervene with students who fall off track. The district also might 
monitor students with many absences, since 77 percent of them eventually dropped out, and just under half of the 
total dropout class was made up of high-absence students. 

The sample charts below illustrate how a district tracked two indicators from freshman year that were both 
predictive and comprehensive— the number of Fs in core courses and the number of credits earned. 



Four-Year Graduation Rate by Number of Core 
Course Fs Freshman Year, Class of 2004 Cohort 



100 % 



Four-Year Graduation Rate by Credits 
Earned Freshman Year, Class of 2004 Cohort 



100 % 
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80% 



60% 



Average for Four-Year 
Graduation 



Point 



40% 



20% 



..SI 



1 2 3 4 5 6 

Number of Semester Fs 



7 8 or 
more 



0 % 

Three 3.5 
or fewer 




Percentage 
of students 



51% 13% 9% 6% 5% 4% 4% 3% 4% 



Percentage 
of freshmen 



4 4.5 5 

Credits Earned Freshman Year 

15% 3% 4% 5% 6% 7% 13% 11% 37% 



Insight 

You can use predictive data to help your district decide how to define "off track." In this case, the number of students who 
eventually drop out is almost the same for two groups of students: those who have two or more Fs and those who earn fewer 
than 5.5 credits in their freshman year. 



Key Steps 

1. After examining chanactenistics fon pnedictiveness 
and compnehensiveness, choose one onsevenal 
chanactenistics that combine to be highly pnedictive and 
compnehensive. 

2. Extend youn on-tnack analysis such that an on-tnack 
indicaton exists fon each yean of school— it may be as 
simple as aligning with cnedits needed each yeanto 
nemain on pace. 

Apply what you have learned from the historical data to one or more classes of current 
students (Class of 2013, Class of 2014, etc.). 

Identify the students in the class of interest, beginning with those who were in your school system in 8th grade. 

Include fields for all the indicators of interest— demographic, academic, and behavioral. If you have to narrow your 
analysis, use the indicators that your historical analyses found to be both predictive and comprehensive. Remove 
students who are not first-time 8th graders. You will need to decide whether to exclude certain other students, such 
as those who are severely developmentally delayed. 



3. Store your on-track indicator as a flag that can be easily 
referenced. 

Key Question 

Some indicators require complex data elements and 
extensive calculations. How do you ensure that an effective 
indicator is well understood? 
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Identify groups of students by indicators of interest. 

Now you have indicators you can use to break down groups of students into even more discrete categories. 

The categories should be based on combinations of predictive and comprehensive indicators. For example, 
group students by age entering school and credits earned. In the chart below, students are grouped into three 
categories: 

Off-Track Students 
by Age and Credits Attained 



Younger, far from graduation: 16 or 17 

years old; two or more years off track for 
graduation 

Older, far from graduation: 18 or older; two 
or more years from graduation 

Older, close to graduation: 18 or older; 
graduation possible within one year 



100 % 
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Old and Close 



20% 



0 % 



Old and Far 



- Young and Far 



Insight 

Knowing how far from graduation 
students of different ages are 
will allow you to customize your 
interventions. An 18-year-old far 
from graduation will need a much 
different approach than an 18-year- 
old only a few credits away. 



Off Track 



Another example: The chart below groups students using performance on the 8th grade English test as one indi- 
cator that is cross-tabbed to the student’s age when entering high school. A key finding from this analysis is that 
nearly one-fourth of students who are proficient in English and arrive in high school at age 13-14 still fall off track. 
This suggests that schools must provide supports for students as they transition to 9th grade and throughout that 
important first year of high school, rather than wait until the end of the year to see who needs help. 

Percentage of All Students Who Are Off Track 

Age at High School Entry 

Over Expected Age (15+) At Expected Age (13-14) 
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5% of Off Track 
(3,000 students) 
enter high school overage but 
with sufficient literacy skills 


Least challenged 
entering high school: 

24% of Off Track 
(16,000 students) 
enter high school on-age with 
sufficient literacy skills 


Most challenged 
entering high school: 

19% of Off Track 
(12,000 students) 
enter high school overage and 
with literacy challenges 


52% of Off Track 
(34,000 students) 
enter high school on-age with 
literacy challenges 



Insight 

Although the majority of students who fall off track during high school enter high 
school with significant, predictable challenges, a large group of students enters 
on-age and nominally prepared but still struggles. 



Key Steps 

1. After at-risk or off-track 
designations have been developed, 
determine groups of interest. 

2. Use data for a single cohort 
of students that tracks 
longitudinally. 

3. Link off-track designation to 
student characteristics, such 
as age at entry or entering test 
scores, to determine breakdown of 
students. 

Key Question 

Does the analysis reveal “quick 
wins," such as the option to provide 
students who are overage but 
proficient with a few extra classes 
to earn credits? 
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Developing Baseline District Facts: Cross- 
Sectional Analysis offers key steps on using exist- 
ing data, such as ethnicity and students receiving 
free and reduced-price lunch, to determine gradua- 
tion rates for those groups. 



Click here to read more about student segmentation 
and the identification of risk indicators in Identifying 
Potential Dropouts, by Craig Jerald for Moving 
Forward: High Standards and High Graduation 
Rates, a joint project of Achieve and Jobs for the 
Future. 



Click here to read about the guidelines the National 
Governors Association has established for creating 
a cohort. The district assigns the expected year of 
graduation (which then identifies the cohort, e.g., 
Class of 2014) as soon as the student first enters 
9th grade. If students transfer in after 9th grade, 
information on their transcript is used to assign 
them to a cohort. 
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Some schools do a better job than others in helping students navigate their way to a 
diploma and postsecondary education. Just as you need to segment your students, 
you need to segment your schools. Some schools and programs might be strong all 
around, meeting or exceeding your goals for all student groups. Some might suc- 
ceed with certain student groups but not with others. With good data, you can share 
the practices of the high performers and deal proactively with struggling schools. 






STRENGTHEN 
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Collect data to show where students who are off track are currently 
enrolled. 

In the example below, most off-track students were enrolled in general high schools, even 
though these schools were less successful than other school models or programs in gradu- 
ating this population of students. 



Enrollment Location of In-School 
Off-Track Students 
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Key Steps 

1. Use a cross-sectional population as shown on 
the chart on page 8 of this report. 

2. Link students to the school attended as of the 
date of analysis (may not necessarily be the 
school used for accountability analyses). 

3. Match schools to the type of model and roll up 
student populations by type of school. 

Key Questions 

1. How does the size of the off-track population 
compare to the number of dedicated seats? 

2. Has the method of serving off-track students 
changed over time? 
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Determine the success of your current mix of schools. 



Use the data to better understand which schools have the highest percentages of students who are on track 
to graduate. 



On-Track Rate by School Type, 2006 



Graduation rate 




System Average 



Insight 

Calculating the performance of individual schools and school models 
without taking into account the types of students served in a school 
can hide significant differences in performance that more complex 
analysis reveals. 



Key Steps 

1. Use either a cross-sectional or longitudinal 
student population, depending on questions to 
be answered. 

2. Determine the outcome metric to be calculated 
(e.g., graduation rate, on-track rate, credits 
accumulated). 

3. Segment schools into model or program type 
and roll up the associated students by type. 

4. Calculate the outcome metric to provide an 
unadjusted view of relative success of different 
options. 

Key Questions 

1. When are students tagged to a school for the 
purposes of this analysis? What happens if a 
student changes schools? 

2. How have outcomes changed over time, 
unadjusted for student population? 



a " _ " _ 

Compare the success of different options. 

In the example below, new small schools were having greater success with 9th graders than larger, compre- 
hensive high schools. The goal is for students to earn 12 credits by the end of freshman year, but they can be 
promoted with as few as seven. 



Credits Earned Freshman Year: 
Entering 1999 and 2004 Freshmen 
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Key Steps 

1. Identify options to compare— in this case, a his- 
torical baseline and then an alternative model to 
the current mainstream system. 

2. Calculate an outcome variable for each indi- 
vidual student included in the analysis. 

Key Question 

Do basic statistics, such as percentage of stu- 
dents being promoted, disguise actual declines in 
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Comprehensive 
High School 



2004 performance? 

New Small 
High School 
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Predict a school's future performance based on early performance. 

This analysis requires development of a basic predictive model that helps 
establish predicted outcomes and interim performance goals forvarious 
options. The chart below illustrates the likelihood of graduation for students 
representing different combinations of age and credits earned. 

Likelihood to Graduate for Age-Credit Combinations 




Credits at Beginning of Year 



Key Steps 

1 . Use a longitudinal student 
population representing a 
single cohort of students. 

2. For each student, determine the 
number of credits earned each 
year and the enrollment status 
at the end of the year [e.g., still 
enrolled, graduate, dropout, 
transfer). 

3. Calculate the graduation rate 
for each combination of age and 
credits earned for each year in 
time. 

Key Questions 

1. When should total credits 
earned be calculated for a 
year? Including summerschool 
credits? 

2. Which graduation rate should 
be used— four-year, five-year, 
seven-year? 



The charts below indicate that small schools are more likely to have higher graduation rates, based on the 
performance of students in Years 1 and 2. 



Small Schools 
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Year 1 Year 2 Year 3 Year 4 Year 5 

0% 0% 1% 42% 47% 



Key Steps 

1. Develop a basic predictive model and as much early data as 
possible that matches the predictive model. 

2. Determine early performance (e.g., freshman credits earned) 
in the comparable options and apply the predictive model that 
translates performance into outcomes. 

3. Establish intermediate outcomes by applying student 
progression in similar models. 



Key Questions 

1 . What variables can be known early in a student’s career that 
increase the power of simple model? 

2. Do similar models exist externally that can provide guidance 
about expected results? 
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Compare outcomes of different programs. 

In the left chart below, participants in the program analyzed were about 50 percent more likely to be on track 
for graduation than those who did not participate. The chart on the right demonstrates the positive impact of 
9th grade transition schools. 



On Track, 2005-06 
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Key Steps 

1. Select programs to be evaluated. 

2. Flag students who participated in the 
programs (the experimental group). 

3. Identify students with similar demographics 
to those who participated— those who chose 
not to participate or are the most similar 
population (the control group). 

4. Choose an outcome to be tested and 
calculate outcomes for both student groups. 

Key Question 

Are there significant differences between 

students who participate in the program and 

those eligible who do not participate? 



Insight 

Modeling expected program performance requires making educated estimates of the performace of new students and expanded 
programs. In choosing these rates, keep in mind that because of a selection bias, additional students may not achieve as signifi- 
cant an improvement as students who opted in to programs early or on their own. 
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Calculate graduation rates for specific student groups. 

In this example from New York City, graduation rates for off-track students in Transfer Schools were nearly three 
times higher than for their peers in large, comprehensive high schools. 



Off-Track Students Graduating 
from Comprehensive High Schools 
& and Recuperative Programs 
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Key Steps 

1. Use a longitudinal student population representing a single cohort 
of students. 

2. Select a student population of interest (e.g., off-track students). 

3. Link students to schools, generally based on where the student 
achieved a final outcome ratherthan based on accountability. 

4. Calculate a graduation rate for different options— likely a longer-term 
graduation rate than fouryears to account for recuperative models. 

Key Questions 

1. Which programs are significantly more successful than traditional 
schools with challenged populations? 

2. Are different models more successful with specific student groups? 



Insight 

To understand the success of recuperative school models, compare similar populations in traditional schools versus other school 
models. Without comparing apples to apples, other school models may appear to perform poorly because they enroll more stu- 
dents with significant challenges. 
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Analyze the effect of changing schools. 



In this chart, again from New York City, students did much better in terms of attendance and credit accumula- 
tion after they enrolled in Transfer Schools. 



Comparative Student Performance: 

High School History Prior to Transfer School 
versus After Enrolling at Transfer School 
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Key Steps 

1. Identify students enrolled in the program or school of interest. 

2. Decide which suspected student performance changes to analyze. 

3. Link students to performance [e.g., attendance, credits earned) 
in the current school and in the prior school. 

4. Calculate the change in performance for the set of students who 
were in a traditional school and moved to a recuperative school. 

Key Questions 

1. How quickly does student performance change after 
transferring? 

2. Are certain schools better at specific aspects of student 
improvement— attendance improvement, credit recovery? 







These slides are accessible online in the Supplement 
to the Analyst Guide: 



Using Multivariate Regression Analysis to 
Calculate Over and Under Performance relies on 
the methodology described in the previous slide to 
create a scatter diagram illustrating school perfor- 
mance falling above or belowthe line. 



Geographical Concentrations of Student Need links 
off-track students to zip codes and then illustrates 
highest concentrations in regions within a district. 

Determining Effect of Structure on Specific 
Students illustrates the success of small schools 
versus other schools in graduating students who 
perform at high and low levels on achievement tests. 

Developing Basic Predictive Models illustrates 
graphically the likelihood of graduation for students 
representing different combinations of age and 
credits earned. 

Predicting Graduation Rates uses weighted aver- 
ages to predict graduation rates for students in 
different schools. 

Advanced Predictions of Performance: Regression 
Analysis describes the use of multiple variables to 
determine which can predict performance. 



Identifying More Effective School Models applies 
information from the advanced predictive model 
to illustrate how different schools support various 
groups of students. 

Understanding Model Variables under District 
Control describes the steps to test the extent to 
which changes, such as fewer students in a school, 
increase the graduation rate. 

Understanding the Effect of Size and Concentration 
on Individual Students shows how to calculate the 
projected graduation rate for a “representative” stu- 
dent in a different school structure, e.g., a school of 
500 versus a school of 3,000. 

Determining Differential Effectiveness illustrates 
the actual (versus predictive) effect of different 
schools on different groups of students. 
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CALCULATE THE 



School reform efforts can be advanced or derailed by one well-timed question: What does it 
cost? Knowing which students could benefit from a different school choice takes you only so 
far in a world of scarce resources and competing agendas. District leaders must know the 
cost of serving students with different needs, the operating costs of different options, and the 
return on investment based on school performance. 



a 



Allocate costs on a per-pupil basis. 

There typically is a wide variation in resource allocation for each type of student 

Sample District, Fully Allocated Cost per Pupil 





Cost per Pupil 


Ratio 


All Students 


$14.3 K 


1.3 


Regular Education 


$10.6 K 


1.0 


Limited English 
Proficient 


$13.6 K 


1.3 


Poverty 


$11.2 K 


1.1 


Students with 
Disabilities (All) 


$28.9 K 


2.7 


Students with 
Disabilities (Resource) 


$24.6 K 


2.3 


Students with 
Disabilities (Self- 
Contained) 


$41.5 K 


3.9 




Key Steps 

1. Merge budget and payroll data, linking salaries with 
benefits, and identify all nonoperating line items and 
code into the budget. 

2. Allocate costs as school specific, central costs, or 
central costs that can be allocated to a specific school to 
determine which costs flow to which students. 

3. Calculate both school-attributed and fully allocated 
weights for each type of student (regular education, 
poverty, mainstream or substantially separate special 
education, and ELL/limited English proficient). 

Key Questions 

1. How are funds allocated on a per-pupil basis? 

2. Is the allocation of funds equitable? 

3. How does this snapshot look across schools? 
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Determine the incremental cost of options. 

Disaggregating the fixed and variable costs of various options will help you determine if you can afford to open new 
schools or programs without closing some current ones. The chart below illustrates per-pupil allocation by groups 
of students; for example, students receiving special education services incur higher per-pupil costs than students 
in regular education. After calculating the district per-pupil cost, it is important to determine the per-pupil costs of 
different groups at each school to see if the allocation by group is equitable from school to school. 



Potential Incremental Costs of Small Schools 




Key Steps 

1. Work closely with the Budget Office to determine baseline total 
costs fon existing tnaditional compnehensive high schools. 

2. Bneak baseline funding down into vaniable costs that move with 
the student if he leaves a school and fixed costs that do not 
move with a student if his old school does not close down. 

3. Calculate the incnemental cost of new options depending on the 
distnict decision whethen a new option will be supplementany on 
will neplace an old school. 

Key Question 

Will new schools neplace old, failing schools, on will new schools 

be supplementany to the cunnent system? 



Key Definitions 

“Cost avoidance" describes savings that occur when students leave one school to attend another. If the old school remains open, 
there may be only partial cost avoidance because the fixed cost of running the old school remains. If the old school closes, there 
is a total cost avoidance of the cost to operate the old schools. 



Use the tools developed by Education Resource Strategies to get a rough estimate of the cost of different options, 
such as rich/lean staffing or smaller versus larger class sizes (www.erstools.org/Dream/index.cfm). You will choose 
options from an abbreviated menu to enter data on the size of your district, the percentage of students in special educa- 
tion, and per-pupil costs. It is easy to try different scenarios in these interactive worksheets, and the approximations 
can help you decide whether to do additional, more specific analysis. 
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Calculate the return-on-investment. 

Return-on-investment calculates the amount of improvement in a performance metric a district can expect given 
a certain value of investment placed against that improvement. These kinds of calculations will help you determine 
the incremental cost of each graduate, with or without school closures. The chart shows that a small school spends 
more per pupil each year but graduates more students— thus the per graduate cost is lower. 





Small Autonomous School 


Comparison Group (Under- 
Performing General High Schools) 


Number of incoming students 
(assumption) 


100 students 


100 students 


Average length of stay in high school 


3.84 years 


3.35 years 


Five-year graduation rate 


73% 


47% 


Approximate Cost per Pupil per Year 


$11,120 


$9,000 


Total cohort cost 


$4.27MM (=100 students x 3.84 years 
x $11,120] 


$3.02MM (= 100 students x 3.35 years 
x $9,000) 


Student outcomes 


64 graduates 


41 graduates 


12 transfers 


12 transfers 


16 dropouts 


43 dropouts 


7 still active after five years 


4 still active after five years 1 


Cost per Graduate 


$66,591 


$73,160 



The chart below illustrates how to calculate cost avoidance: the savings that occur when students leave one school 
to attend another. If the former school remains open, then there may be only partial cost avoidance given the fixed 
costs of operating that school. If the school closes, then there is total cost avoidance of operating costs. 



Potential “cost avoidance” of new 
schools 


$9,000 per pupil (with closures) 


$3,000 per pupil (no closures) 


Total incremental cost 


$1.25 MM (with closures) 


$3.26MM (no closures) 


New graduates 


23 graduates 


Incremental Cost per New Graduate 


$55K (with closures) 


$143K (no closures) 



Key Steps 

1. Finalize cost and performance figures: 

Graduation rate (including transfers) 

Cost per student 
Length of time enrolled 

2. Calculate cost per graduate: (length of stay) X (cost per 
year) / graduation rate 



3. Finalize cost avoidance figures. 

4. Calculate incremental cost per graduate: (cost of new 
option - cost of old option) / new graduates produced 

Key Question 

Can the district afford to open new options without closing 
down other schools? 
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TO DRIVE 









The leaders in your school district will rely on the data generated by these analyses to make 
decisions on how to intervene to support struggling students and how to prioritize available 
resources. There are few decisions more important than how to serve students and use 



taxpayer dollars wisely. At least one member of the data team should be at the table to 
present findings, respond to questions, and ensure that conclusions are warranted, whether 
the table is in the superintendent's conference room or in the school board meeting room. 
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The data tools in this guide inform steps in a long- 
term process that includes stakeholder engage- 
ment, organization design, and policy alignment. 



If the data are credible and presented in a format 
that is easy to understand, district leaders can 
make the case for change to families and stu- 
dents who have the most to gain — or lose— from 
those decisions. 



If those who work in portfolio development offices 
can get their hands on the return on investment 
for each school in the portfolio, they will be able 
to make judicious decisions on how to support 
those schools. 



And if leaders and school board members adopt 
policies that take data into account, the result is 
more thoughtful and instructive direction for district 
action. 







Information for this report is drawn from 
Atlanta, Boston, Charlotte-Mecktenburg, 
Chicago, Dallas, Portland, and New York City 
and their partnerships with the Bridgespan 
Group, Boston Consulting Group, McKinsey 
& Company, Education Resource Strategies, 
and The Parthenon Group. 
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